netty的direct memory大小设置

2023-07-02 01:52| 来源: 网络整理| 查看: 265

最近遇到一个 netty 的 OutOfDirectMemoryError 报错，是在分配 direct memory 时内存不足导致的，看了下报错提示，要分配的内存大小为 16M，剩余的空间不足。这里 max direct memory 大约有 7G，于是就有一个疑问，这个值是怎么设置的？

代码分析

这里使用的 netty 版本是 4.1.14.Final，如下是报错时的调用栈信息，主要关注下 PlatformDependent 这个类。

1 2 3 4 5 6 7 8 9 10 11 Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 7532970287, max: 7549747200) at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:618) at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:572) at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181) at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)

找到 PlatformDependent 的第 572 行，位于 allocateDirectNoCleaner 函数内，它的功能是根据指定的容量（capacity）分配一个新的 ByteBuffer

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 /** * Allocate a new {@link ByteBuffer} with the given {@code capacity}. {@link ByteBuffer}s allocated with * this method MUST be deallocated via {@link #freeDirectNoCleaner(ByteBuffer)}. */ public static ByteBuffer allocateDirectNoCleaner(int capacity) { assert USE_DIRECT_BUFFER_NO_CLEANER; // 第572行 // 增加已使用内存计数，若内存不足，直接抛出异常 incrementMemoryCounter(capacity); try { // 分配内存 return PlatformDependent0.allocateDirectNoCleaner(capacity); } catch (Throwable e) { // 分配失败，减小已使用内存计数 decrementMemoryCounter(capacity); throwException(e); return null; } }

查看第 572 行对应的 incrementMemoryCounter 实现，它的功能是增加已使用内存的计数

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 private static void incrementMemoryCounter(int capacity) { if (DIRECT_MEMORY_COUNTER != null) { for (;;) { // 获取当前已使用内存计数 long usedMemory = DIRECT_MEMORY_COUNTER.get(); // 计算新的已使用内存计数 long newUsedMemory = usedMemory + capacity; // 超过了最大的容量限制，抛出异常 if (newUsedMemory > DIRECT_MEMORY_LIMIT) { throw new OutOfDirectMemoryError("failed to allocate " + capacity + " byte(s) of direct memory (used: " + usedMemory + ", max: " + DIRECT_MEMORY_LIMIT + ')'); } // CAS更新计数值 if (DIRECT_MEMORY_COUNTER.compareAndSet(usedMemory, newUsedMemory)) { break; } } } }

从代码逻辑可见，它是通过 CAS 更新已使用内存计数。在更新前先判断是否超过了 DIRECT_MEMORY_LIMIT 最大容量限制，若已超过则直接抛出异常，也就是说此时并未真正地分配内存。

这里就有个问题，DIRECT_MEMORY_LIMIT 是怎么设置的？

搜索代码发现，它是在 PlatformDependent 的静态代码块中设置的，代码如下：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 // Here is how the system property is used: // // * < 0 - Don't use cleaner, and inherit max direct memory from java. In this case the // "practical max direct memory" would be 2 * max memory as defined by the JDK. // * == 0 - Use cleaner, Netty will not enforce max memory, and instead will defer to JDK. // * > 0 - Don't use cleaner. This will limit Netty's total direct memory // (note: that JDK's direct memory limit is independent of this). // 默认未设置，所以maxDirectMemory值为-1 long maxDirectMemory = SystemPropertyUtil.getLong("io.netty.maxDirectMemory", -1); if (maxDirectMemory == 0 || !hasUnsafe() || !PlatformDependent0.hasDirectBufferNoCleanerConstructor()) { USE_DIRECT_BUFFER_NO_CLEANER = false; DIRECT_MEMORY_COUNTER = null; } else { USE_DIRECT_BUFFER_NO_CLEANER = true; if (maxDirectMemory < 0) { // 取值逻辑在这里 maxDirectMemory = maxDirectMemory0(); if (maxDirectMemory 0，不使用清理器（cleaner），表示 netty 的最大 direct memory 限制

它的默认值是 -1，根据代码逻辑会执行到 maxDirectMemory = maxDirectMemory0() 这行，该方法的实现如下：

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 private static final Pattern MAX_DIRECT_MEMORY_SIZE_ARG_PATTERN = Pattern.compile( "\\s*-XX:MaxDirectMemorySize\\s*=\\s*([0-9]+)\\s*([kKmMgG]?)\\s*$"); private static long maxDirectMemory0() { long maxDirectMemory = 0; ClassLoader systemClassLoader = null; try { // 1. 通过反射调用sun.misc.VM.maxDirectMemory() systemClassLoader = getSystemClassLoader(); Class vmClass = Class.forName("sun.misc.VM", true, systemClassLoader); Method m = vmClass.getDeclaredMethod("maxDirectMemory"); maxDirectMemory = ((Number) m.invoke(null)).longValue(); } catch (Throwable ignored) { // Ignore } if (maxDirectMemory > 0) { return maxDirectMemory; } try { // 2. 通过MBean获取-XX:MaxDirectMemorySize配置，因为Android没有这些类，所以使用反射获取 Class mgmtFactoryClass = Class.forName( "java.lang.management.ManagementFactory", true, systemClassLoader); Class runtimeClass = Class.forName( "java.lang.management.RuntimeMXBean", true, systemClassLoader); Object runtime = mgmtFactoryClass.getDeclaredMethod("getRuntimeMXBean").invoke(null); @SuppressWarnings("unchecked") List vmArgs = (List) runtimeClass.getDeclaredMethod("getInputArguments").invoke(runtime); for (int i = vmArgs.size() - 1; i >= 0; i --) { Matcher m = MAX_DIRECT_MEMORY_SIZE_ARG_PATTERN.matcher(vmArgs.get(i)); if (!m.matches()) { continue; } maxDirectMemory = Long.parseLong(m.group(1)); switch (m.group(2).charAt(0)) { case 'k': case 'K': maxDirectMemory *= 1024; break; case 'm': case 'M': maxDirectMemory *= 1024 * 1024; break; case 'g': case 'G': maxDirectMemory *= 1024 * 1024 * 1024; break; } break; } } catch (Throwable ignored) { // Ignore } if (maxDirectMemory

【本文地址】

公司简介

联系我们